Electronic apparatus and controlling method thereof

ABSTRACT

An electronic apparatus is provided. The electronic apparatus includes a camera; a memory configured to store at least one instruction; and at least one processor configured to execute the at least one instruction to: detect at least one object included in an image captured by the camera; identify information on an engagement of each of the at least one object with the electronic apparatus; obtain gesture information of each of the at least one object; obtain a target object from among the at least one object based on an operation status of the electronic apparatus, the information on the engagement of each of the at least one object, and the obtained gesture information of each of the at least one object; identify a function corresponding to gesture information of the target object; and execute the identified function.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2019-0149440 filed on Nov. 20, 2019,in the Korean Intellectual Property Office, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an electronic apparatus and a controllingmethod thereof and more particularly, to an electronic apparatus thatprovides a function corresponding to a user gesture based on anoperation status of the electronic apparatus, a user's engagementregarding the electronic apparatus, and user gesture information, and acontrolling method thereof.

2. Description of Related Art

With the development of electronic technology, gesture recognitiontechnology has been applied to various industries. Recently, gesturerecognition technology has been applied to various electronic devicesand applications so that, for example, a user may manipulate a smartphone or a tablet PC through a gesture.

An electronic apparatus where related art gesture technology is appliedoperates such that while maintaining an interaction with a first user byrecognizing a gesture of the first user and executing a functioncorresponding to the intention of the first user, if a gesture of asecond user is recognized, the electronic apparatus may stop theinteraction with the first user while the interaction of the first useris not completed and execute a function corresponding to the gesture ofthe second user. In this case, even though the second user does notintend to execute a function of the electronic apparatus, the electronicapparatus may recognize the gesture of the second user as a commandgesture for executing a specific function and thus, execute a functionthat is not intend by the user.

When recognizing a gesture and executing a function in an environmentwhere there is a plurality of users, there may be problem that afunction is executed regardless of the intention of a user, or a user'sgesture is misrecognized and a wrong function is executed, thusdeteriorating the convenience and satisfaction of the user whomanipulates an electronic apparatus using a gesture.

Therefore, there is a need for natural interaction technology capable ofaccurately identifying a user's intention for manipulating an electronicapparatus and executing a function corresponding to the user'sintention.

SUMMARY

Provided is an electronic apparatus capable of performing a naturalinteraction with a user by executing a function corresponding to theuser's intention.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, an electronic apparatusincludes: a camera; a memory configured to store at least oneinstruction; and at least one processor configured to execute the atleast one instruction to: detect at least one object included in animage captured by the camera; identify information on an engagement ofeach of the at least one object with the electronic apparatus; obtaingesture information of each of the at least one object; obtain a targetobject from among the at least one object based on an operation statusof the electronic apparatus, the information on the engagement of eachof the at least one object, and the obtained gesture information of eachof the at least one object; identify a function corresponding to gestureinformation of the target object; and execute the identified function.

The processor may be further configured to execute the at least one toidentify the information on the engagement of each of the at least oneobject based on at least one of a distance from the electronic apparatusto each of the at least one object, an angle at which each of the atleast one object gazes at the electronic apparatus, a time at which eachof the at least one object gazes at the electronic apparatus, anddirection information regarding a body and head of each of the at leastone object.

The processor may be further configured to execute the at least oneinstruction to: obtain gesture information of each of the at least oneobject based on an image obtained during a plurality of frames; obtainan average value and a variance value of the angle at which each of theat least one object, obtained during the plurality of frames, gazes atthe electronic apparatus; and identify the information on the engagementof each of the at least one object based on the average value and thevariance value.

The processor may be further configured execute the at least oneinstruction to: obtain candidate objects having an engagement greaterthan a predetermined engagement from among the at least one object; andidentify, from among the candidate objects, a candidate object making agesture corresponding to a predetermined gesture as the target object.

The processor may be further configured execute the at least oneinstruction to: obtain candidate objects having an engagement greaterthan a predetermined engagement from the at least one object; andidentify, from among the candidate objects, a candidate object at ashortest distance from the electronic apparatus as the target object.

An operation status of the electronic apparatus may include a firstoperation status in which an input of a wake-up gesture is awaited, asecond operation status in which interaction with an object isperformed, and a third operation status in which interaction with anobject is blocked, wherein the memory is configured to store informationindicating whether a function corresponding to a gesture type isexecuted based on an operation status of the electronic apparatus andinformation regarding the gesture type, and wherein the processor may befurther configured to execute the at least one instruction to: identifythe operation status of the electronic apparatus and whether thefunction corresponding to the gesture type of the target object isexecuted during the operation status, and execute the function based ona result of the identification.

The processor may be further configured execute the at least oneinstruction to execute a function corresponding to gesture informationof the target object only when the gesture of the target object is thefirst gesture type that induces entry into the second operation statusduring the first operation status.

The processor may be further configured execute the at least oneinstruction to, based on a gesture type of the target objectcorresponding to a second gesture type pointing in a predetermineddirection during the second operation status, execute a functioncorresponding to gesture information of the target object when anengagement of the target object is less than a predetermined engagement.

The processor may be further configured execute the at least oneinstruction to not execute a function corresponding to gestureinformation of the target object during the third operation status.

In accordance with an aspect of the disclosure, a method of controllingan electronic apparatus, the controlling method includes: obtaining animage; detecting at least one object included in the obtained image;identifying information on an engagement of each of the at least oneobject with the electronic apparatus; obtaining gesture information ofeach of the at least one object; obtaining a target object from amongthe at least one object based on an operation status of the electronicapparatus, the information on the engagement of each of the at least oneobject, and the obtained gesture information of each of the at least oneobject; identifying a function corresponding to gesture information ofthe target object; and executing the identified function.

The identifying information on the engagement of each of the at leastone object may include identifying the information on the engagementbased on at least one of a distance from the electronic apparatus toeach of the at least one object, an angle at which each of the at leastone object gazes at the electronic apparatus, a time at which each ofthe at least one object gazes at the electronic apparatus, and directioninformation regarding a body and head of each of the at least oneobject.

The obtaining gesture information may include obtaining gestureinformation of each of the at least one object based on an imageobtained during a plurality of frames, and the identifying informationon the engagement of each of the at least one object may includeobtaining an average value and a variance value of the angle at whicheach of the at least one object, obtained during the plurality offrames, gazes at the electronic apparatus, and identifying theinformation on the engagement of each of the at least one object basedon the average value and the variance value.

The obtaining a target object may include obtaining candidate objectshaving an engagement greater than a predetermined engagement from the atleast one object, and identifying, from among the candidate objects, acandidate object making a gesture corresponding to a predeterminedgesture as the target object.

The obtaining the target object may include obtaining candidate objectshaving an engagement greater that predetermined engagement from the atleast one object, and identifying, from among the candidate objects, acandidate object at a shortest distance from the electronic apparatus asthe target object.

An operation status of the electronic apparatus may include a firstoperation status in which an input of a wake-up gesture is awaited, asecond operation status in which interaction with an object isperformed, and a third operation status in which interaction with anobject is blocked, wherein the identifying the function may include:storing information indicating whether a function corresponding to agesture type is executed based on an operation status of the electronicapparatus and information regarding the gesture type; identifying theoperation status of the electronic apparatus and whether the functioncorresponding to the gesture type of the target object is executedduring the operation status; and executing the function based on aresult of the identification.

The executing the function may include executing a functioncorresponding to gesture information of the target object only when thegesture of the target object is the first gesture type that inducesentry into the second operation status during the first operationstatus.

The executing the function may include, based on a gesture type of thetarget object corresponding to the second gesture type pointing in apredetermined direction during the second operation status, executing afunction corresponding to gesture information of the target object whenan engagement of the target object is less than a predeterminedengagement.

A function corresponding to gesture information of the target object maynot executed during the third operation status.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is an environmental view of an electronic apparatus according toan embodiment;

FIG. 2 is a block diagram showing a configuration of an electronicapparatus according to an embodiment;

FIG. 3 is a flowchart showing an operation of an electronic apparatusaccording to an embodiment;

FIG. 4 is a diagram showing an operation of obtaining engagementinformation of an electronic apparatus according to an embodiment;

FIG. 5 is a diagram showing an operation of obtaining a target object ofan electronic apparatus according to an embodiment;

FIG. 6 is a table indicating whether a function of an electronicapparatus is executed according to an embodiment;

FIG. 7 shows a user being viewed from a perspective of an electronicapparatus according to an embodiment;

FIG. 8 is a flowchart showing method of identifying an operation statusof an electronic apparatus according to an embodiment;

FIG. 9 is a diagram showing an operation of an electronic apparatusaccording to an embodiment; and

FIG. 10 is a flowchart showing a controlling method of an electronicapparatus according to an embodiment.

DETAILED DESCRIPTION

The terms used in the present disclosure will be descried briefly, andthe present disclosure will be described in detail.

General terms that are currently widely used are selected as terms usedin embodiments of the disclosure in consideration of functions in thedisclosure, but may be changed depending on the intention of thoseskilled in the art or a judicial precedent, the emergence of a newtechnique and the like. In addition, in a specific case, termsarbitrarily chosen by an Applicant may exist. In this case, the meaningsof such terms are mentioned in detail in corresponding descriptionportions of the disclosure. Therefore, the terms used in the embodimentsof the disclosure need to be defined on the basis of the meanings of theterms and the contents throughout the disclosure rather than simplenames of the terms.

The disclosure may be variously modified and have several embodiments,and thus specific embodiments of the disclosure are shown in thedrawings and described in detail in the detailed description. However,the disclosure is not limited to specific embodiments, but includes allmodifications, equivalents and substitutions without departing from thescope and spirit of the disclosure. If it is decided that a detaileddescription for the known art related to the disclosure may obscure thegist of the disclosure, the detailed description is omitted.

Terms ‘first’, ‘second’ and the like, may be used to describe variouscomponents, but the components are not to be construed as being limitedby the terms. The terms are used only to distinguish one component fromanother component.

Singular forms used herein are intended to include plural forms unlessexplicitly indicated otherwise. The terms ‘comprise’ or ‘include’ usedin the specification, specify the presence of features, numerals, steps,operations, components, parts mentioned in the present specification orcombinations thereof, but do not preclude the presence or addition ofone or more other features, numerals, steps, operations, components,parts or combinations thereof.

Although the terms first, second, third, etc., may be used herein todescribe various elements, but elements are not limited by these terms.These terms are only used to distinguish one element from anotherelement. For example, without departing from the scope of thedisclosure, a first element may be termed as a second element, and asecond element may be termed as a first element. The term of “and/or”includes a plurality of combinations of relevant items or any one itemamong a plurality of relevant items.

Hereinafter, embodiments of the disclosure are described in detail withreference to the accompanying drawings so that those skilled in the artto which the present disclosure pertains may easily practice thedisclosure. However, the disclosure may be implemented in variousdifferent forms and is not limited to the embodiments described herein.In addition, in the drawings, portions unrelated to the description areomitted to obviously describe the disclosure, and similar portions aredenoted by similar reference numerals throughout the specification. Therelative size and depiction of these elements are not necessarily toscale and may be exaggerated for clarity, illustration, and convenience.

FIG. 1 shows an electronic apparatus according to an embodiment.Specifically, FIG. 1 shows a situation in which there are a plurality ofobjects in a Field of View (FOV) of an electronic apparatus 100.

In the situation of FIG. 1 , a related art gesture recognitionapparatus, when recognizing a gesture of a second object 2 that is anon-user while performing an interaction with a first object 1 that is auser, stops the interaction with the first object 1 and executes afunction corresponding to the gesture of the second object 2.Accordingly, user convenience and satisfaction is deteriorated.

The electronic apparatus 100 according to an example embodiment maydetect the first object 1 and the second object 2, and obtain gestureinformation and engagement regarding each of the first object 1 and thesecond object 2. Here, the engagement means a value indicating thedegree to which an object is engaged in an electronic apparatus. Forexample, the engagement may be obtained based on whether an object isgazing at the electronic apparatus.

The electronic apparatus 100 may obtain a target object based on atleast one of the obtained gesture information, engagement, or operationstatus of the electronic apparatus 100, and may identify a functioncorresponding to the gesture information of the target object. Forexample, the electronic apparatus 100 may obtain the first object 1 as atarget object based on the obtained engagement. Specifically, if theengagement of the first object 1 is greater than predeterminedengagement, the electronic apparatus 100 may obtain the first object 1as the target object. In this case, the electronic apparatus 100 mayobtain gesture information of the first object 1, and perform aninteraction with the first object 1 based on the obtained gestureinformation. For example, the electronic apparatus 100 may execute afunction corresponding to the gesture information of the first object 1.In addition, even if gesture information of the second object 2 isobtained while performing an interaction with the first object 1, theelectronic apparatus 100 may not execute a function corresponding to thegesture information of the second object 2. Alternatively, if the firstobject 1 is obtained as the target object, the electronic apparatus 100may not obtain the gesture information of the second object 2. In otherwords, the electronic apparatus 100 may perform the interaction with thefirst object 1 continuously. Accordingly, user convenience andsatisfaction can be improved.

According to another embodiment, the electronic apparatus 100 may obtaina target object based on obtained gesture information and engagement.For example, if the first object 1 makes a wake-up gesture and hasengagement greater than predetermined engagement, the electronicapparatus 100 may obtain the first object 1 as the target object.

According to another embodiment, the electronic apparatus 100 may obtaina target object based on the operation status of the electronicapparatus 100. For example, the electronic apparatus 100 may obtain thefirst object 1 as the target object while performing an interaction withthe first object 1. In this case, even if the second object 2 hasengagement greater than predetermined engagement and makes a wake-upgesture, the electronic apparatus 100 may not obtain the second object 2as the target object.

The electronic apparatus 100 may perform a function corresponding togesture information of an object based on the operation status of theelectronic apparatus 100 and the gesture type of the object. Forexample, if the electronic apparatus 100 is in a standby status waitingfor a user's wake-up gesture and the first object 1 makes a pointinggesture that does not correspond to the wake-up gesture, the electronicapparatus 100 may not perform a function corresponding to the pointinggesture even if the engagement of the first object 1 is greater thanpredetermined engagement. This is because if the electronic apparatus100 reacts to a gesture other than the wake-up gesture in the standbystatus and executes a function corresponding to the gesture, the usermay be inconvenienced since the function that is not intended by theuser is executed.

As such, the electronic apparatus 100 may execute a functioncorresponding to a gesture of an object based on not only the engagementof the object but also the operation status of the electronic apparatus100 and the gesture type. Accordingly, the user convenience andsatisfaction can be improved.

In the above, the operation of an electronic apparatus according to anembodiment has been described. However, this is only an example, andother embodiments of the electronic apparatus will be described later.

FIG. 2 is a block diagram illustrating a configuration of an electronicapparatus according to an embodiment. In particular, the electronicapparatus 100 may be a movable robot, and as illustrated in FIG. 2 , mayinclude a camera 110, a display 120, a communication interface 130, amemory 140, a processor 150, a sensor 160, and a travelling unit 170.The electronic apparatus 100 does not necessarily have to include all ofthe above components, and some of the components may be omitted orimplemented.

The camera 110 may obtain an image by photographing the surroundings ofthe electronic apparatus 100. In addition, the camera 110 may be varioustypes of cameras. For example, the camera 110 may be one of a 2D-basedred-green-blue (RGB) camera or an infrared (IR) camera. Alternatively,the camera 110 may be one of a 3D-based Time of Flight (ToF) camera or astereo camera.

The display 120 may display various images. For example, if theelectronic apparatus 100 is a retail robot, the display 120 may displayinformation regarding a product provided in a space in which theelectronic apparatus 100 exists. However, this is only an example, andthe display 120 may display various images such as an image thatrequires a user's input, a feedback regarding a user input, etc. Thedisplay 120 may be a touch screen. In this case, the processor 150 mayobtain a user's touch input through the display 120.

The communication interface 130 may include at least one circuit, andmay perform communication with various types of external devices. Forexample, the communication interface 130 may perform communication withan external server or a user terminal. In addition, the communicationinterface 130 may perform communication with an external device usingvarious types of communication methods. The communication interface 130may perform data communication wirelessly or via a cable. When thecommunication interface 130 performs communication with an externaldevice through a wireless communication method, the communicationinterface 130 may include at least one of a WiFi communication module, acellular communication module, a 3G (3rd generation) mobilecommunication module, a 4G (4th generation) mobile communication module,a 4G Long Term Evolution (LTE) communication module, or a 5G (5^(th)generation) mobile communication module. The communication interface 130may be implemented as a wired communication module (e.g., LAN, etc.).

The memory 140 may store an operating system (OS) for controlling theoverall operations of the components of the electronic apparatus 100 andinstructions of data related to the components of the electronicapparatus 100. To this end, the memory 140 may be implemented as anon-volatile memory (e.g., hard disk, Solid status drive (SSD), or flashmemory), a volatile memory, etc.

For example, the memory 140 may store an instruction that, when an imageis obtained from the camera 110, causes the processor 150 to obtaingesture information regarding an object included in the image. Inaddition, the memory 140 may store a neural model for recognizing agesture of an object. In particular, the neural network model may beexecuted by an existing general-purpose processor (e.g., CPU) or aseparate AI-only processor (e.g., GUP, NPU, etc.).

The processor 150 may control the overall operations of the electronicapparatus 100. For example, the processor 150 may detect at least oneobject included in an image captured by the camera 110. In addition, theprocessor 150 may obtain and identify information on engagement of atleast one object regarding the electronic apparatus 100.

The processor 150 may obtain information on engagement based on at leastone of the distance from the electronic apparatus 100 to the object, theangle at which the object gazes at the electronic apparatus 100, ordirection information of the body and head of the object.

In addition, the processor 150 may obtain gesture information of anobject based on an image obtained by the camera 100 during a pluralityof frames. In this case, the processor 150 may obtain an average valueand a variance value of an angle at which the object obtained during aplurality of frames gazes at the electronic apparatus 100. Further, theprocessor 150 may obtain the engagement of the object based on theobtained average value and variance value. For example, if the obtainedaverage value is less than a predetermined average value, and theobtained variance value is less than a predetermined variance value, theprocessor 150 may identify that the object is engaged in the electronicapparatus 100.

The processor 150 may obtain one of at least one object as a targetobject based on the operation status of the electronic apparatus 100,information on engagement of the object, and obtained gestureinformation.

The processor 150 may obtain candidate objects that are identified asengaged in the electronic apparatus 100 from among at least one object,and obtain a candidate object that makes a gesture corresponding to apredetermined gesture from among the candidate objects as a targetobject.

In addition, the processor 150 may obtain a candidate object located atthe closest distance from the electronic apparatus 100 out of thecandidate objects identified as engaged in the electronic apparatus 100as a target object.

The processor 150 may identify a function corresponding to gestureinformation of a target object. In this case, the processor 150 mayidentify a function corresponding to gesture information of a targetobject based on the operation status of the electronic apparatus 100,information on engagement of the object, and obtained gestureinformation. The processor 150 may execute the identified function.

The memory 140 may match and store the operation status of theelectronic apparatus 100, information regarding a gesture type, andwhether a function corresponding to the gesture type is executed. Inthis case, the processor 150 may identify the operation status of theelectronic apparatus 100 and whether a function corresponding to thegesture type of the target object is executed during the operationstatus.

The operation status of the electronic apparatus 100 may include a firstoperation status in which the electronic apparatus 100 waits for aninput of a wake-up gesture. In other words, the first operation statusmay be a standby status.

In addition, the operation status of the electronic apparatus 100 mayinclude a second operation status in which the electronic apparatus 100performs an interaction with an object. In this case, the secondoperation status may include a user command status in which theelectronic apparatus 100 performs an interaction with an object inresponse to a command of the object, such as a user. For example, theuser command status may include a status in which the electronicapparatus 100 provides information regarding a product to a user throughthe display 120 in response to the user's product guide command.According to another embodiment, the user command status may include astatus in which the electronic apparatus 100 moves toward the user inresponse to the user's call command. According to yet anotherembodiment, the user command status may include a status in which theelectronic apparatus 100 guides the user to a route in response to theuser's route guide command. However, the present disclosure is notlimited to the above-described embodiments, and may include a status inwhich the electronic apparatus 100 performs an interaction with a userin response to the user's various commands.

In addition, the second operation status may include an active triggerstatus in which the electronic apparatus 100 performs an interactionwith an object regardless of a command of the object. For example, theactive trigger status may include a status in which the electronicapparatus 100 approaches and greets a user.

Further, the operation status of the electronic apparatus 100 mayinclude a third operation status in which an interaction with an objectis blocked. For example, the third operation status may include a statusin which the electronic apparatus returns to a charging station forcharging and a status in which the electronic apparatus 100 moves to apredetermined place in an emergency situation such as a fire.

The processor 150 may execute a function corresponding to gestureinformation of the target object only when the gesture type of thetarget object in the first operation status corresponds to the firstgesture type that induces entry into the second operation status. Here,the first gesture type may mean a wake-up gesture. Accordingly, when theprocessor 150 obtains gesture information of the target objectcorresponding to the first gesture type in the first operation status,the electronic apparatus 100 may enter the second operation status fromthe first operation status.

In addition, when the gesture type of the target object in the secondoperation status corresponds to a second gesture type pointing in apredetermined direction, the processor 150 may execute a functioncorresponding to the gesture information of the target object even ifthe engagement of the target object is less than predeterminedengagement. This is because, when the target object gives a command tothe electronic apparatus 100 through the pointing gesture by pointing ina predetermined direction, the target object may not gaze at theelectronic apparatus 100. In addition, the processor 150 may not executea function corresponding to the gesture information of the target objectin the third operation status.

The sensor 160 may be configured to detect information regarding thesurrounding environment of the electronic apparatus 100. In particular,the sensor 160 may include a proximity sensor for detecting an approachby a user or an object, a gyro sensor for obtaining motion information,an acceleration sensor, etc. The sensor 160 may include a sensor forobtaining biometric information of a user, and a sensor for obtaininginformation (e.g., temperature, humidity, etc.) regarding an environmentwhere the electronic apparatus 100 is located.

The travelling unit 170 may be configured to move the electronicapparatus 100. In particular, the travelling unit 170 may include anactuator for driving the electronic apparatus 100. In addition, otherthan the travelling unit 170, an actuator for driving the motion ofother physical components (e.g., an arm, etc.) of the electronicapparatus 100 may be included. For example, the electronic apparatus 100may control an actuator to travel or move in the direction of a senseduser through the sensor 160.

Hereinafter, an operation of the electronic apparatus 100 will bedescribed with reference to FIG. 3 .

FIG. 3 is a flowchart showing an operation of an electronic apparatusaccording to an embodiment.

Referring to FIG. 3 , the electronic apparatus 100 may obtain an image.For example, the electronic apparatus 100 may obtain an image which iscaptured by the camera 110 for a plurality of frames.

In addition, the electronic apparatus 100 may detect at least one objectincluded in the obtained image (operation S310). In this case, theelectronic apparatus 100 may detect or recognize at least one object byinputting the obtained image to a neural network model that is trainedto detect or recognize an object included in an image.

Further, the electronic apparatus 100 may obtain engagement informationof an object (operation S320). In this case, the electronic apparatus100 may obtain or identify the engagement of the object detected inoperation S310 regarding the electronic apparatus 100. The detaileddescription regarding how to obtain engagement information will bedescribed later with reference to FIG. 4 .

The electronic apparatus 100 may obtain gesture information of an object(operation S330). In this case, the electronic apparatus 100 may obtaingesture information of the object detected in operation S310. Thegesture information of an object may include various types of gestures.For example, the gesture information may include the first gesture typethat is a wake-up gesture and the second gesture type that is a pointinggesture pointing in a predetermined direction. In addition, the gestureinformation may include the third gesture type indicating the status oremotion of an object, such as a user. In this case, the third gesturetype may be an expressive gesture. For example, the expressive gesturemay include a gesture in which a user requests a high-five or handshakefrom the electronic apparatus 100. In addition, the gesture informationmay include the fourth gesture type that is a symbolic gesture. Examplesof the fourth gesture type include a gesture for describing the shape orsize of an object by a user's hand, a gesture for indicating a numberwith a finger, and a gesture for indicating a language such as a signlanguage.

The electronic apparatus 100 may obtain gesture information using aneural network model. For example, the electronic apparatus 100 mayobtain gesture information of an object included in an obtained image byinputting the obtained image to a neural network model which is trainedto obtain gesture information of an object included in an image. In thiscase, the electronic apparatus 100 may obtain gesture information of theobject by inputting multi-frame images captured continuously at apredetermined time interval to the neural network model. In addition,the electronic apparatus 100 may obtain gesture information of theobject using an object detection and object tracking algorithm.

The electronic apparatus 100 may obtain the operation status of theelectronic apparatus 100 (operation S340). In this case, the electronicapparatus 100 may obtain the operation status in various ways. Forexample, the electronic apparatus 100 may obtain the operation statusstored in the memory 140. In addition, the electronic apparatus 100 mayobtain behavior information including a behavior pattern by analyzingthe behavior of the object. In this case, the electronic apparatus 100may identify the operation status of the electronic apparatus 100 basedon the behavior information of the object. For example, when theelectronic apparatus 100 obtains the first behavior information where adetected object approaches the electronic apparatus 100, the electronicapparatus 100 may predict that the operation status of the electronicapparatus 100 is an active trigger status.

The electronic apparatus 100 may obtain a target object from multipleobjects based on at least one of information on engagement of theobject, gesture information and the operation status of the electronicapparatus 100 obtained in operations S320, S330 and S340, respectively.For example, the electronic apparatus 100 may obtain a target objectbased on information regarding engagement of the object and gestureinformation. In this case, the electronic apparatus 100 may obtain anobject taking a wake-up gesture with engagement greater thanpredetermined engagement from among at least one object as a targetobject. According to another embodiment, the electronic apparatus 100may obtain a target object based on the gesture information of theobject and the operation status of the electronic apparatus 100. In thiscase, the electronic apparatus 100 may obtain an object taking a gesturecorresponding to the second gesture type in the second operation statusfrom among at least one object as a target object.

The electronic apparatus 100 may identify a function corresponding tothe gesture information of the object (operation S350). Specifically,the electronic apparatus 100 may identify a function corresponding tothe gesture information of the target object. In addition, theelectronic apparatus 100 may execute the identified function (S360).

Hereinafter, an operation of obtaining engagement information of anelectronic apparatus will be described with reference to FIG. 4 .

FIG. 4 shows an operation of obtaining engagement information of anelectronic apparatus according to an embodiment.

Referring to FIG. 4 , the electronic apparatus 100 may identifyengagement of a user 4 regarding the electronic apparatus 100 based onan angle (θ_(gaze)) at which the user 4 gazes at the electronicapparatus 100. Specifically, the electronic apparatus 100 may obtain acoordinate value (x_(f), y_(f), z_(f)) for the center position of theface of the user 4. In this case, the electronic apparatus 100 mayobtain the coordinate value (x_(f), y_(f), z_(f)) based on depthinformation included in the image capturing the face of the user 4.

In addition, the electronic apparatus 100 may obtain a gaze angle(θ_(gaze)) based on the following Equation (1).

$\begin{matrix}{\theta_{gaze} = {\cos^{- 1}\frac{\overset{\rightarrow}{v_{f}} \cdot \overset{\rightarrow}{v_{h}}}{{\overset{\rightarrow}{v_{f}}}\mspace{11mu}{\overset{\rightarrow}{v_{h}}}}}} & (1)\end{matrix}$

Here {right arrow over (v_(f))}=−(x_(f), y_(f), z_(f)), {right arrowover (v_(h))}=(x_(h), y_(h), z_(h)), {right arrow over (v_(f))} is avector in the direction of the electronic apparatus from the center ofthe face of the user 4, and {right arrow over (v_(h))} is a head posevector of the user 4.

In addition, the electronic apparatus 100 may identify whether or notthe user 4 is engaged, based on the obtained gaze angle (θ_(gaze)) andEquation (2).

$\begin{matrix}{E_{T} = \left\{ \begin{matrix}{1,} & {{{if}\mspace{14mu}{{Avg}._{T}\left( \theta_{gaze} \right)}} < {{th}_{a}\mspace{14mu}{and}\mspace{14mu}{{Var}._{T}\left( \theta_{gaze} \right)}} < {th}_{v}} \\{0,} & {otherwise}\end{matrix} \right.} & (2)\end{matrix}$

Here E_(T) is an engagement value, Avg_(˜T)(θ_(gaze)) sis the averagevalue of the gaze angle (θ_(gaze)) th_(a) is a predetermined averagevalue, Var_(·T)(θ_(gaze)) is the variance value of the gaze angle(θ_(gaze)), th_(v) is a predetermined variance value, and T is a timesection where the gesture of the user 4 is recognized.

The electronic apparatus 100 may identify that the user 4 is engaged inthe electronic apparatus 100 only when the average value(Avg_(˜T)(θ_(gaze)) of the gaze angle (θ_(gaze)) is less than thepredetermined average value (th_(a)), and the variance value(Var_(·T)(θ_(gaze))) of the gaze angle (θ_(gaze)) is less than thepredetermined variance value (th_(v)) (i.e., E_(T)=1). In other cases,the electronic apparatus 100 may identify that the user 4 is not engagedin the electronic apparatus 100 (i.e., E_(T)=0). In another embodiment,the engagement (E_(T)) may not have a value of 0 or 1, and may havevarious values. For example, the engagement value of the user 4 may be0.6, and the predetermined engagement value may be 0.5. In this case,since the engagement value of the user 4 is greater than thepredetermined engagement value, the electronic apparatus 100 mayidentify or determine that the user 4 is engaged in the electronicapparatus 100.

The electronic apparatus 100 may determine engagement based on the gazeangle of a user as described above, but this is only an example. Theelectronic apparatus 100 may identify or determine the engagement of theuser 4 in various ways. According to another embodiment, the electronicapparatus 100 may identify the engagement of the user 4 based on thetime during which the user 4 gazes at the electronic apparatus 100. Forexample, if the time during which the user 4 gazes at the electronicapparatus 100 is greater than a predetermined time, the electronicapparatus 100 may identify the engagement value (E_(T)) to be 1. In thiscase, the electronic apparatus 100 may identify that the user 4 isengaged in the electronic apparatus 100. According to another exampleembodiment, the electronic apparatus 100 may identify the engagement ofthe user 4 based on a pose vector of each of the body and face of theuser 4. For example, if a difference value between the pose vector ofthe body of the user 4 and the pose vector of the face of the user 4 isless than a predetermined value, the electronic apparatus 100 mayidentify the engagement value (E_(T)) to be 1. In this case, theelectronic apparatus 100 may identify that the user 4 is engaged in theelectronic apparatus 100. In addition, the electronic apparatus 100 maytrack the eyeball(s) of the user 4 to identify whether the user 4 gazesat the electronic apparatus 100. For example, if the angle between thecentral axis of the eyeball of the user 4 and the central axis of theFOV of the camera 110 is within a predetermined angle range, theelectronic apparatus 100 may identify the engagement value (E_(T)) to be1.

Hereinafter, a method of obtaining a target object of an electronicapparatus will be described with reference to FIG. 5 .

FIG. 5 shows an operation of obtaining a target object of an electronicapparatus according to an embodiment. Specifically, FIG. 5 is a top viewof an electronic apparatus and objects.

As shown in FIG. 5 , a plurality of objects may exist in the FOV of theelectronic apparatus 100. In this case, the electronic apparatus 100 mayobtain information regarding engagement of each of the plurality ofobjects. In addition, the electronic apparatus 100 may obtain a firstcandidate object 5 and a second candidate object 6 having an engagementvalue greater than predetermined engagement value from among theplurality of objects. In this case, the electronic apparatus 100 mayobtain the first candidate object 5 that makes a gesture correspondingto a predetermined gesture between the first candidate object 5 and thesecond candidate object 6 as a target object. The predetermined gesturemay be a gesture corresponding to the above-described first gesturetype.

The electronic apparatus 100 may obtain a target object based on theoperation status of the electronic apparatus 100, gesture information ofan object, and information regarding engagement, and execute a functioncorresponding to the gesture information of the target object.

FIG. 6 is a table showing whether an object is engaged and whether afunction of an electronic apparatus is executed according to anembodiment.

As shown in FIG. 6 , whether a function of the electronic apparatus isexecuted may be obtained based on the operation status of an electronicapparatus, the gesture type of an object, information regardingengagement. The data indicating whether a function of the electronicapparatus is executed may be stored in the memory 140 of the electronicapparatus 100.

Hereinafter, the operation of an electronic apparatus according tovarious cases shown in FIG. 6 will be described.

In the first operation status, the electronic apparatus 100 may executea function corresponding to the gesture information of the object onlywhen obtaining gesture information corresponding to the first gesturetype. In this case, when gesture information of a type other than thefirst gesture type is obtained, even though it is identified that theobject is engaged in the electronic apparatus 100, the electronicapparatus 100 may not execute a function corresponding to the gestureinformation. In other words, the electronic apparatus 100 may notexecute a function corresponding to a gesture other than the wake-upgesture in the first operation status.

FIG. 7 shows a user 7 being viewed from a perspective of the electronicapparatus 100 according to an embodiment. In FIG. 7 , the user 7 maymake a pointing gesture corresponding to the second gesture type withoutgazing at the electronic apparatus 100. In this case, if the electronicapparatus is performing an interaction with the user 7, even though theuser 7 does not gaze at the electronic apparatus 100, the electronicapparatus 100 may execute a function corresponding to the gesture of theuser 7. Accordingly, the user 7 may continue to interact with theelectronic apparatus 100.

As such, even if it is identified that the object is not engaged in theelectronic apparatus 100, when the gesture information of the objectcorresponds to the second gesture type, the electronic apparatus 100 mayexecute a function corresponding to the gesture information of theobject. Referring back to FIG. 6 , in the second operation status, eventhough the engagement of the object is less than predeterminedengagement, the electronic apparatus 100 may execute a functioncorresponding to the gesture of the object as long as the gesture of theobject corresponds to the second gesture type. That is, in the secondoperation state, when the gesture of the object is identified as thesecond gesture type, the electronic device 100 may execute a functioncorresponding to the gesture of the object irrespective of whether theobject is engaged in the electronic apparatus 100. For example, when auser makes a gesture pointing to a specific product with a hand whilelooking at the specific product, in the second operation state, theelectronic device 100 a function corresponding to the user's gesture(for example, a function of providing information on a product indicatedby a user) even if the user does not stare at the electronic device 100.Accordingly, user satisfaction and convenience can be improved.

In the active trigger status, if the gesture information of the objectcorresponds to the third gesture type or the fourth gesture type, theelectronic apparatus may not execute a function corresponding to thegesture information of the object even if the engagement of the objectis greater that predetermined engagement. For example, if the electronicapparatus 100 performs the operation of greeting a user who approachesthe electronic apparatus 100 regardless of a user's command, theelectronic apparatus 100 may not perform a function corresponding to thegesture of the user even if the user makes a gesture corresponding tothe fourth type, such as a hand gesture indicating a number while gazingat the electronic apparatus 100. This is because the user usually doesnot make a gesture corresponding to the fourth gesture type when theelectronic apparatus 100 is in the active trigger status. In this case,the user may be provided with a function from the electronic apparatusby making a gesture corresponding to the first gesture type such as awake-up gesture and then, making a gesture corresponding to the thirdgesture type or the fourth gesture type.

In the third operation status, the electronic apparatus 100 may notexecute a function corresponding gestures of all gesture types of theobject regardless whether the object is engaged or not. This is because,in the third operation status such as an emergency situation, it may notpossible for the electronic apparatus 100 to perform an interaction witha user (e.g., when charging is required), or performing a predeterminedoperation (e.g., moving to a place for charging) would be more importantthan performing an interaction with a user.

Hereinafter, the method of identifying an operation status of anelectronic apparatus will be described.

FIG. 8 shows a method of identifying an operation status of anelectronic apparatus according to an embodiment. As described above, theelectronic apparatus 100 may execute a function corresponding to thegesture information of a target object based on the operation status ofthe electronic apparatus 100.

Referring to FIG. 8 , the electronic apparatus 100 may detect an object(operation S810). When an object is detected, the electronic apparatus100 may obtain an identification value corresponding to the detectedobject. The electronic apparatus 100 may store the obtainedidentification value in the memory 140. This operation may correspondingto operation S310 of FIG. 3 above and thus, description overlapping withFIG. 3 will be omitted.

The electronic apparatus 100 may identify whether the identificationvalue of the detected object is recorded in the memory 140 (operationS820). If the identification value regarding the detected object beforethe detection of the object is recorded in the memory 140, theelectronic apparatus 100 may identify whether the command history of thedetected object is recorded in the memory 140 (operation S830). In thiscase, if the command history of the detected object is recorded in thememory 140, the electronic apparatus 100 may identify that the operationstatus of the electronic apparatus 100 is the user command status.Alternatively, the command history of the object is not recorded in thememory 140, the electronic apparatus may identify that the operationstatus of the electronic apparatus 100 is the active trigger status. Ifthe identification value regarding the detected object before thedetection of the object is not recorded in the memory 140, theelectronic apparatus 100 may identify whether the electronic apparatus100 is on the move (operation S840). In this case, if the electronicapparatus 100 is on the move, the electronic apparatus 100 may identifythat the operation status of the electronic apparatus 100 is the thirdoperation status. If the electronic apparatus 100 is not on the move,the electronic apparatus 100 may identify that the operation status ofthe electronic apparatus 100 is the first operation status.

As such, based on the identified operation status, the electronicapparatus 100 may execute a function corresponding to the gestureinformation of the target object.

The operation status of the electronic apparatus 100 may changeaccording to various conditions. For example, when the electronicapparatus 100 in the first operation status obtains a trigger gesture ofa user, the electronic apparatus 100 may enter the user command status.In this case, the operation status of the electronic apparatus 100 maybe converted from the first operation status to the user command status.If the user's gesture input is not received for a predetermined time,the electronic apparatus 100 in the user command status may enter thefirst operation status again. In this case, the operation status of theelectronic apparatus may be converted from the user command status tothe first operation status.

FIG. 9 shows an operation of an electronic apparatus according to anembodiment.

Referring to FIG. 9 scenario (a), the electronic apparatus 100 mayperform an interaction with at least one of a plurality of users in anenvironment where the plurality of users exist. For example, theelectronic apparatus 100 may perform an interaction with a first user 9.Specifically, when obtaining a wake-up gesture of the first user 9having engagement greater than predetermined engagement, the electronicapparatus 100 in the first operation status may enter the secondoperation status. Here, the second operation status may mean the usercommand status. In this case, the electronic apparatus 100 may obtainthe first user 9 as a target object, and perform an interaction with thefirst user 9 based on an additional gesture of the first user 9. Inaddition, in the user command status, even if a second user 10 havingengagement greater than predetermined engagement within the FOV of theelectronic apparatus 100 makes the gesture of calling the electronicapparatus 100, the electronic apparatus 100 may perform an interactionwith the first user 9 without executing a function corresponding to thegesture of the second user 10. Therefore, the electronic apparatus 100may provide the first user 9 who is the target object with a functioncorresponding to the gesture information of the first user 9continuously. Accordingly, the satisfaction of the first user 9 can beimproved.

Referring to FIG. 9 scenario (b), if a gesture of the first user 9 isnot obtained for a predetermined time from the time when the electronicapparatus 100 enters the second operation status, the electronicapparatus may enter the first operation status. If a gesture in whichthe second user 10 having engagement greater than predeterminedengagement calls the electronic apparatus 100 is obtained, theelectronic apparatus 100 may obtain the second user 10 as a targetobject and move near the second user 10.

FIG. 10 is a flowchart describing a controlling method of an electronicapparatus according to an embodiment.

Referring to FIG. 10 , the controlling method may include obtaining animage (operation S1010), detecting at least one object included in theimage (operation S1020), determine information on engagement of the atleast one object regarding an electronic apparatus (operation S1030),obtaining gesture information of the at least one object (operationS1040), obtaining a target object from among at least one object basedon the operation status of the electronic apparatus, the information onengagement, and the obtained gesture information, and identifying afunction corresponding to the gesture information of the target object(operation S1050), and executing the identified function (operationS1060).

Hereinafter, each operation will be described.

The electronic apparatus 100 may obtain an image (operation S1010). Inthis case, the electronic apparatus 100 may obtain an image whichcaptures the periphery of the electronic apparatus 100 by the camera110.

The electronic apparatus 100 may detect at least one object included inthe image (operation S1020). In this case, the electronic apparatus 100may obtain at least one of position information, type information, ordistance information of the object. In addition, the electronicapparatus 100 may obtain an identification value regarding the detectedobject. This operation may correspond to operation S310 of FIG. 3described above.

The electronic apparatus 100 may identify information on engagementregarding the electronic apparatus by at least one object (operationS1030). In this case, the electronic apparatus 100 may identifyengagement of the at least one object detected at S1010 regarding theelectronic apparatus. In addition, the electronic apparatus 100 maydetermine whether the at least one object is engaged in the electronicapparatus 100 based on the obtained engagement. Operation S1030 maycorrespond to operation S320 of FIG. 3 described above and thus,overlapping description will be omitted.

In addition, the electronic apparatus 100 may obtain gesture informationof the at least one object (operation S1040). In this case, theelectronic apparatus 100 may obtain gesture information of an objectincluded in the image by analyzing the obtained image. In addition, theelectronic apparatus 100 may obtain images of multiple frames which arecaptured continuously and obtain gesture information of the object.Operation S1040 may correspond to operation S330 of FIG. 3 describedabove.

In addition, the electronic apparatus 100 may obtain a target objectfrom among at least one object based on the operation status of theelectronic apparatus 100, the information on engagement and the obtainedgesture information and identify a function corresponding to the gestureinformation of the target object (operation S1050). The presentoperation may be understood by those skilled in the art to correspond tooperations S350, S360 of FIG. 3 and FIG. 6 and thus, detaileddescription thereof will be omitted.

In addition, the electronic apparatus 100 may execute the identifiedfunction (operation S1060).

The embodiments described above may be implemented in a computer or anapparatus similar to the computer using software, hardware, or acombination of software and hardware. In some cases, embodimentsdescribed in the disclosure may be implemented by a processor itself.According to a software implementation, embodiments such as proceduresand functions described in the specification may be implemented byseparate software modules. Each of the software modules may perform oneor more functions and operations described in the disclosure.

Computer instructions for performing processing operations according tothe embodiments described above may be stored in a non-transitorycomputer-readable medium. The computer instructions stored in thenon-transitory computer-readable medium cause a specific device toperform the processing operations of the display apparatus according tothe diverse embodiments described above when they are executed by aprocessor of the specific device.

The non-transitory computer-readable medium is not a medium that storesdata for a while, such as a register, a cache, a memory, or the like,but means a medium that semi-permanently stores data and is readable bythe device. Specific examples of the non-transitory computer-readablemedium may include a compact disk (CD), a digital versatile disk (DVD),a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, and the like.

While embodiments of the disclosure have been shown and described, thedisclosure is not limited to the aforementioned embodiments, and it isapparent that various modifications can be made by those having ordinaryskill in the art to which the disclosure belongs, without departing fromthe gist of the disclosure as claimed by the appended claims, and suchmodifications are not to be interpreted independently from the technicalidea or prospect of the disclosure.

What is claimed is:
 1. An electronic apparatus comprising: a camera; amemory configured to store at least one instruction; and at least oneprocessor configured to execute the at least one instruction to: detecta plurality of objects included in an image captured by the camera;obtain information on an engagement of the plurality of objects with theelectronic apparatus; obtain gesture information of the plurality ofobjects; obtain candidate objects having an engagement greater than apredetermined engagement from among the plurality of objects; identify,from among the candidate objects, a candidate object making a gesturecorresponding to a predetermined gesture as the target object; identifya function corresponding to gesture information of the target object;and execute the identified function.
 2. The electronic apparatus asclaimed in claim 1, wherein the at least one processor is furtherconfigured to execute the at least one instruction to obtain theinformation on the engagement of the plurality of objects based on atleast one of a distance from the electronic apparatus to the pluralityof objects, an angle at which the plurality of objects gazes at theelectronic apparatus, a time at which the plurality of objects gazes atthe electronic apparatus, and direction information regarding a body andhead of the plurality of objects.
 3. The electronic apparatus as claimedin claim 2, wherein the at least one processor is further configured toexecute the at least one instruction to: obtain gesture information ofthe plurality of objects based on an image obtained during a pluralityof frames; obtain an average value and a variance value of the angle atwhich the plurality of objects, obtained during the plurality of frames,gazes at the electronic apparatus; and obtain the information on theengagement of the plurality of objects based on the average value andthe variance value.
 4. The electronic apparatus as claimed in claim 1,wherein the at least one processor is further configured execute the atleast one instruction to: identify, from among the candidate objects, acandidate object at a shortest distance from the electronic apparatus asthe target object.
 5. The electronic apparatus as claimed in claim 1,wherein the operation status of the electronic apparatus comprises afirst operation status in which an input of a wake-up gesture isawaited, a second operation status in which interaction with an objectis performed, and a third operation status in which interaction with theobject is blocked, wherein the memory is configured to store informationindicating whether a function corresponding to a gesture type isexecuted based on the operation status of the electronic apparatus andinformation regarding the gesture type, and wherein the at least oneprocessor is further configured to execute the at least one instructionto: identify at least one operation status of the electronic apparatusand whether the function corresponding to the gesture type of the targetobject is executed during the operation status, and execute the functionbased on a result of the identification of the operation status.
 6. Theelectronic apparatus as claimed in claim 5, wherein the at least oneprocessor is further configured execute the at least one instruction toexecute a function corresponding to gesture information of the targetobject only when the gesture of the target object is a first gesturetype that induces entry into the second operation status during thefirst operation status.
 7. The electronic apparatus as claimed in claim5, wherein the at least one processor is further configured execute theat least one instruction to, based on a gesture type of the targetobject corresponding to a second gesture type pointing in apredetermined direction during the second operation status, execute afunction corresponding to gesture information of the target object whenan engagement of the target object is less than a predeterminedengagement.
 8. The electronic apparatus as claimed in claim 5, whereinthe at least one processor is further configured execute the at leastone instruction to not execute a function corresponding to gestureinformation of the target object during the third operation status.
 9. Amethod of controlling an electronic apparatus, the controlling methodcomprising: obtaining an image; detecting a plurality of objectsincluded in the obtained image; obtaining information on an engagementof the plurality of objects with the electronic apparatus; obtaininggesture information of the plurality of objects; obtaining candidateobjects having an engagement greater than a predetermined engagementfrom among the plurality of objects; identifying, from among thecandidate objects, a candidate object making a gesture corresponding toa predetermined gesture as the target object; identifying a functioncorresponding to gesture information of the target object; and executingthe identified function.
 10. The method as claimed in claim 9, whereinthe obtaining information on the engagement of the plurality of objectscomprises obtaining the information on the engagement based on at leastone of a distance from the electronic apparatus to the plurality ofobjects, an angle at which the plurality of objects gazes at theelectronic apparatus, a time at which the plurality of objects gazes atthe electronic apparatus, and direction information regarding a body andhead of the plurality of objects.
 11. The method as claimed in claim 10,wherein the obtaining gesture information comprises obtaining gestureinformation of the plurality of objects based on an image obtainedduring a plurality of frames, and wherein the obtaining information onthe engagement of the plurality of objects comprises obtaining anaverage value and a variance value of the angle at which each of theplurality of objects, obtained during the plurality of frames, gazes atthe electronic apparatus, and obtaining the information on theengagement of each of the plurality of objects based on the averagevalue and the variance value.
 12. The method as claimed in claim 9,wherein the obtaining the target object comprises obtaining candidateobjects having an engagement greater that predetermined engagement fromthe plurality of objects, and identifying, from among the candidateobjects, a candidate object at a shortest distance from the electronicapparatus as the target object.
 13. The method as claimed in claim 9,wherein the operation status of the electronic apparatus comprises afirst operation status in which an input of a wake-up gesture isawaited, a second operation status in which interaction with an objectis performed, and a third operation status in which interaction with theobject is blocked, wherein the identifying the function comprises:storing information indicating whether a function corresponding to agesture type is executed based on the operation status of the electronicapparatus and information regarding the gesture type; identifying theoperation status of the electronic apparatus and whether the functioncorresponding to the gesture type of the target object is executedduring the operation status; and executing the function based on aresult of the identification of the operation status.
 14. The method asclaimed in claim 13, wherein the executing the function comprisesexecuting a function corresponding to gesture information of the targetobject only when the gesture of the target object is a first gesturetype that induces entry into the second operation status during thefirst operation status.
 15. The method as claimed in claim 13, whereinthe executing the function comprises, based on a gesture type of thetarget object corresponding to a second gesture type pointing in apredetermined direction during the second operation status, executing afunction corresponding to gesture information of the target object whenan engagement of the target object is less than a predeterminedengagement.
 16. The method as claimed in claim 13, wherein a functioncorresponding to gesture information of the target object is notexecuted during the third operation status.